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[57] ABSTRACT 

Neural network algorithms have impressively demon- 
strated the capability of modelling spatial information. 
On the other hand, the application of parallel distrib- 
uted models to processing of temporal data has been 
severely restricted. The invention introduces a novel 
technique which adds the dimension of time to the well 
known back-propagation neural network algorithm. In 
the space-time neural network disclosed herein, the 
synaptic weights between two artificial neurons (pro- 
cessing elements) are replaced with an adaptable-adjust- 
able filter. Instead of a single synaptic weight, the in- 
vention provides a plurality of weights representing not 
only association, but also temporal dependencies. In this 
case, the synaptic weights are the coefficients to the 
adaptable digital filters. 


42 Claims, 9 Drawing Sheets 



Xjln> 



5,253,329 

Page 2 


OTHER PUBLICATIONS 
“Reading the Structure of Brains”, Network Computa- 
tion in Neural Systems, vol. 1, No. 1, Jan. 1980, V. Brai- 
tenberg. 

“Neural Networks Primer Part VI”, AI Expert, pp. 
41-46, 1989, Miller Freeman Publications, San Fran- 
cisco, Calif., Maureen Caudill. 

“The Tempo-Algorithm: Learning in a Neural Net- 
work with Variable Time-Delays”, JJCNN-90-Wash, 
D.C., Jan. 15-19, 1990, vol. 1, pp. I-597-I-600, Ulrich 
Bodenhausen. 

“Temporal-Spatial Coding Transformation: Conver- 
sion of Frequency-Code to Place-Code via a Time-De- 
layed Neural Network”, IJCNN-90 proceedings Jan. 
1990, pp. I— 1 30— I— 1 33, David C. Tam. 


“A Connectionist Approach to the Processing of Time 
Dependent Medical Parameters” IJCNN-90 proceed- 
ings Jan. 1990, pp. II— 575— II— 578, Barry Blumeenfeld, 
MD. 

“Spatio-Temporal vs. Spatial Pattern Recognition by 
the Neocognitron”, IJCNN-90 proceedings Jan. 1990, 
pp. II— 279— II— 282, Kunihiko Fukushima. 
“Self-Organizing Neural Architectures for Motion Per- 
ception, Adaptive Sensory-Motor Control, and Asso- 
ciative Mapping”, DCNN-90 proceedings Jan. 1990, 
pp. II— 26— II— 29, Stephen Grossberg. 

“Adaptive Junction: A Spatio-Temporal Neuron 
Model”, Proceedings, International Joint Conference 
on Neural Networks, Washington, D.C., Jan. 1990, vol. 
1, pp. I— 353— I— 356, Yoshiaki Ajioka, et al. 




U.S. Patent 


Oct. 12, 1993 


Sheet 1 of 9 


5,253,329 




U.S. Patent 


Oct. 12, 1993 


Sheet 2 of 9 


5,253,329 


FIG.3 INPUT LAYER 



FIG.4 




U.S. Patent 


Oct. 12, 1993 


Sheet 3 of 9 



FI6.5A FIG.5B 



2 

S-PLANE Z-PLANE 


FIG.6A 














PjCSj(n)) 


U.S. Patent 


Oct. 12, 1993 


Sheet 5 of 9 


5,253,329 



FI6.8 













U.S. Patent 


Oct. 12, 1993 


Sheet 7 of 9 


5,253,329 


FIG.II 


F(x) = 7t* sin(x) 





U.S. Patent Oct. 12, 1993 sheet 8 of 9 5,253,329 


o 







FIG.I3 


U.S. Patent 


Oct. 12 , 1993 


Sheet 9 of 9 


5,253,329 



00 



1 


5,253,329 


NEURAL NETWORK FOR PROCESSING BOTH 
SPATIAL AND TEMPORAL DATA WITH TIME 
BASED BACK-PROPAGATION 

ORIGIN OF THE INVENTION 

The invention described herein was made by employ- 
ees of the United States Government and ma be manu- 
factured and used by or for the Government of the 
United States of America for governmental purposes 
without payment of any royalties thereon or therefor. 

BACKGROUND OF THE INVENTION 

The present invention relates to a neural network for 
processing both spacial and temporal data (hereinafter 
“space-time neural network”) and to an artificial neu- 
ron, or so-called “processing element”, for use in such a 
space-time neural network. 

More particularly, the invention relates to a space- 
time neural network, and a processing element therefor, 
which receives a temporal sequence of inputs X(n), 
X(n— 1), X(n — 2) ...» where each input X(n) is com- 
prised of N components xi(n), X2(n), . . . x/n), . . . XA<n), 
and which maps such input representations into a single, 
plural-component output representation. The network 
may be a single layer network or it may comprise multi- 
ple layers of processing elements. 

HISTORICAL PERSPECTIVE 

Throughout history, the meaning of time has plagued 
the minds of mankind. The wise Greek philosophers, 
Socrates, Plato and Aristotle, pondered deeply about 
the influence of time had on human knowledge. The 
English poet, Ben Johnson, wrote “O for an engine to 
keep back all clocks”, giving voice to our ageless la- 
ment over the brevity of human life. The great scientist 
Einstein, who developed the theory of relativity, be- 
lieved that space and time cannot be considered sepa- 
rately, but that they depend upon one another. 

There is an urgent need for systems which will reli- 
ably capture space-time knowledge. Human cognitive 
thought processes involve the use of both space and 
time. A childhood event is remembered by an occur- 
rence (or space) and its associated place in time. We 
speak of an event which occurred a specific time ago. 
Linguistic meanings are expressed in a manner in which 
proper temporal order plays a crucial role in the con- 
veyance of a concept. Take, for example, the phrases 
“house cat” and “cat house”. Speech production, too, is 
very order dependent— subtleties in intonations may 
change the whole meaning of a concept. The more 
advanced engineering systems have characteristics 
which vary over time. For instance, complex machines 
such as the Space Shuttle Main Engine abound with 
sensors, each having characteristics which vary over 
the life of the machine’s operation. A system which is 
capable of automatically associating spatial information 
with its appropriate position in time becomes increas- 
ingly significant in our age of automation. 

Also, microscopic level investigations reveal a need 
to incorporate time or sequence discovery and adapta- 
tion into the neuron modelling framework. It is clearly 
evident that information exchange at the neuronal level 
occurs through a rich interchange of complex signals. 
Extensive research has been done on the olfactory bulb 
at anatomical, physiological, and behavioral levels. See 
W. J. Freeman, “Why Neural Networks Don’t Yet Fly: 
Inquiry into the Neurodynamics of Biological Intelli- 


2 

gence” IEEE International Conference on Neural Net- 
works, San Diego, Calif., 1988, and B. Baird, Nonlinear 
Dynamics of Pattern Formation and Pattern Recognition 
in the Rabbit Olfactory Bulb , Elsevier Science Publish- 
5 ers B. V., North-Holland Physics Publishing Division, 
0167-2789, 1986. These research findings have shown 
that information in biological networks takes the form 
of space-time neural activity patterns. The dynamic 
space-time patterns encode past experience, attempt to 
1° predict future actions, and are unique to each biological 
network. 

The Neuron 

As seen in FIG. 1, the “classical” biological neuron 
15 has several dendrites which receive information from 
other neurons. The soma or cell body performs a wide 
range of functions; it processes information from the 
dendrites in a manner which is not entirely understood 
and also maintains the cell’s health. The information 
20 processed by the neuron is distributed by its axon to 
other interconnected neurons by the propagation of a 
spike or action potential. Along each dendrite are thou- 
sands of protrusions where neurons exchange informa- 
25 tion through a region known as the “synapse”. The 
5 synaptic cleft releases chemicals called “neurotransmit- 
ters”. Even at this microscopic level, the relevance for 
time-adaptive neural networks becomes clearly evident. 
Synaptic clefts take on various modifying roles such as 
3Q neurotransmitter modulators, generators, and filters 
which cloud the neuron’s inner workings and render 
these ever-changing dynamical properties especially 
difficult to study. 

Connectionist architectures have impressively dem- 
35 onstrated several models of capturing temporal and 
spatial knowledge. To accomplish this, the most popu- 
lar solution has been to distribute a temporal sequence 
by forcing it into a spatial representation. This method 
has worked well in some instances. See, e.g., J. A. Vil- 
40 larreal and P. Baffes, “Sunspot Prediction Using Neural 
Networks”, SOAR ’89 — Third Annual Workshop on 
Automation and Robotics, 1987. But there are insur- 
mountable problems with this approach and it has ulti- 
mately proven inadequate as a general technique. 

45 Review of Neural Networks 

A neural network is comprised of numerous, indepen- 
dent, highly interconnected artificial neurons, hereinaf- 
ter called “processing elements”, which simulate the 
50 functions of biological neurons. For so-called “back- 
propagation networks”, each element can be character- 
ized as having some input connections from other pro- 
cessing elements and some output connections to other 
elements. The basic operation of a processing element is 
55 to compute its activation value based upon its inputs and 
to send that value to its output. FIG. 2 is a schematic 
diagram of such a processing element. Note that this 
element has j input connections coming from j input 
processing elements. Each connection has an associated 
60 value called a “weight”. The output of this processing 
element is a non-linear transform of its summed, con- 
tinuous-valued inputs by the so-called “sigmoid trans- 
formation”, as discussed in D. E. Rumelhart et al. 
“Learning Internal Representations by Error Propaga- 
65 tion”, in D. E. Rumelhart & J. L. McClelland (Eds.), 
Parallel Distributed Processing: Explorations in the Micro- 
structure of Cognition (Vol. 1) (pp. 318-362) MIT Press, 
1986, Cambridge, Mass. 
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When groups of such processing elements are ar- 
ranged in sequential layers, each layer interconnected 
with the subsequent layer, the result is a wave of activa- 
tions propagated from the input processing elements, 
which have no incoming connections, to the output 
processing elements. The layers of elements between 
the inputs and outputs take on intermediate values 
which perform a mapping from the input representation 
to the output representation. It is from these intermedi- 
ate or “hidden” elements that the back-propagation 
network draws its generalization capability. By forming 
transformations through such intermediate layers, a 
backpropagation network can arbitrarily categorize the 
features of its inputs. 


Ei = Xwijpj 


Pi - /(£,) - 


l 
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( 1 ) 

( 2 ) 


The Weights Of A Back-propagation Network 

The heart of the back-propagation algorithm lies in 
how the values of its interconnections, or weights, are 
updated. Initially, the weights in the network are set to 
some small random number to represent no association 
between processing elements. Upon being given a set of 
patterns representing pairs of input/output associations, 
the network enters what is called a “training phase”. 
During training, the weights are adjusted according to 
a learning algorithm, such as that described by Rumel- 
hart et al. The training phase is modelled after a behav- 
ioristic approach which operates through reinforce- 
ment by negative feedback. That is, the network is 
given an input from some input/output pattern for 
which it generates an output by propagation. Any dis- 
crepancies found when comparing the network’s output 
to the desired output constitute mistakes which are then 
used to alter the network characteristics. According to 
the Rumelhart et al. technique, every weight in the 
network is adjusted to minimize the total mean square 
errors between the response of the network, P p/, and the 
desired outputs, t p i, to a given input pattern. First, the 
error signal, 6/, is determined for the output layer, N: 

0) 

The indices p and i represent the pattern number and 
the index to a node respectively. The weights are ad- 
justed according to: 

A wif n + 1 ) = a A *,/") -f tjS/" + Dp/") (4) 

where Aw gOO is the error gradient of the weight from 
the j-th processing element in layer n to the i-th unit in 
the subsequent layer (n+ 1). The parameter a performs 
a damping effect through the multi-dimensional error 
space by relying on the most recent weight adjustment 
to determine the present adjustment. The overall effect 
of this weight adjustment is to perform a gradient de- 
scent in the error space; however, note that true gradi- 
ent descent implies infinitesimally small increments. 
Since such increments would be impractical, is used to 
accelerate the learning process. In general, then, the 
errors are recursively back propagated through the 
higher layers according to: 


8<") = 1 8j n+ I) *$ l) P'(£f°) 

5 where P'(E) is the first derivative of P(E). 

Other Spatiotemporal Neural Network Architectures 

A number of significant advances in capturing spatial- 
temporal knowledge with neural network systems have 
^ been made to date. See, e.g., M. I. Jordan “Serial Order: 
A Parallel Distributed Processing Approach”, ICS Re- 
port 8604, Institute for Cognitive Science, University of 
California, San Diego, Calif., 1986, and J. L. Elman 
“Finding Structure in Time”, CRL Technical Report 
5 8801, Center for Research in Language, University of 
California, San Diego, Calif., 1988. Jordan approaches 
this problem by partitioning the input layer in a connec- 
tionist network into separate plan and state layers. In 
essence, Jordan’s architecture acts as a back-propaga- 
tion network, except for the specialized processing ele- 
ments in the state layer, which receive their inputs from 
the output units, as well as from recurrent connections 
which allow the state layer elements to “remember” the 
25 network’s most recent state. In other words, the state 
units behave as “pseudo inputs” to the network provid- 
ing a past-state history. Here, a recurrent connection is 
one in which it is possible to follow a path from a pro- 
cessing element back onto itself as shown in FIG. 3. 
30 Recurrent networks of this type allow the element’s 
next state to be not only dependent on external inputs, 
but also upon the state of the network at its most previ- 
ous time step. In general, however, this network is 
trained to reproduce a predetermined set of sequence 
35 patterns from a static input pattern. 

As an example, this network architecture has been 
used by J. A. Villareal, one of the named inventors 
herein, in developing a speech synthesizer. The inputs 
to the speech synthesis network represented a tri- 
4Q phoneme combination and the output was partitioned to 
represent the various vocal tract components necessary 
to produce speech. Thus, the output layer in the speech 
synthesis neural network consisted of the coefficients to 
a time-varying digital filter, a gain element, and a pitch 
45 element which excited the filter, and a set of down- 
counting elements where each count represented a 100 
millisecond speech segment. To train a single tri-phone 
set, the network was first reset by forcing the activation 
value of the processing elements in the state layer to 
50 zero. A tri-phone pattern was then presented to the 
network’s input and held there during the learning pro- 
cess while the outputs changed to produce the appropri- 
ate output characteristics for that particular tri-phone 
combination. The outputs would represent the transi- 
55 tion from one phoneme to another while a smooth tran- 
sition in pitch, gain, and vocal tract characteristics 
would take place. The process was then repeated for 
other tri-phone combinations. 

As shown in FIG. 4 , Elman has modified Jordan’s 
60 approach by constructing a separate layer, called the 
“context layer”, which is equal in size to the number of 
units in the hidden layer. In this network the context 
units receive their inputs along a one-to-one connection 
from the hidden units, instead of from the output units 
65 as described by Jordan. The network works as follows: 
Suppose there is a sequential pattern to be processed. 
Initially, the activation values in the context units are 
reset to a value midway between the upper and lower 
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bounds of a processing element’s activation value, indi- 
cating ambiguous or “don’t care” states. A pattern is 
presented to the network’s input, forward propagating 
the pattern toward the output. At this point, the hidden 
layer activation levels are transferred one-for-one to 5 
elements in the context layer. If desired, error back- 
propagation learning can now take place by adjusting 
the weights between output and hidden, hidden and 
input, and hidden and context layers. The recurrent 
connections from the hidden to context layers are not io 
allowed to change. At the next time step, the network’s 
previous state is encoded by the activation levels in the 
context units. Thus, the context layer provides the net- 
work with a continuous memory. 

SUMMARY OF THE INVENTION 
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and temporal data, the network being adapted to re- 
ceive a sequence of inputs X(n), X(n— 1), X(n— 2). . . , 
each input X(n) being comprised of N components 
xi(n), X 2 (n), . . . x/n), . . . x//(n). The network comprises 
the combination of: 

(a) a plurality L of first processing elements, each first 
processing element (i) comprising a plurality N of 
adaptable filters (Fj/, F 2 /, . . . F )/, . . . F #/), each filter F;/ 
having an input for receiving a respective component 
x/n), x/n— 1), x/n— 2), . . . , of the sequence of inputs, 
where x/n) is the most current input component, and 
providing a filter output y/n) in response to an input 
x/n) which is given by: 

bkjxfi,n-k)\ 


A principal object of the present invention is to pro- 
vide a processing element for a space-time neural net- 
work which is capable of processing temporal as well as 
spacial data. 20 

A further principal object of the invention is to pro- 
vide a space-time neural network comprised of a plural- 
ity of the aforementioned processing elements, which is 
capable of processing temporal as well as spacial data. 

These objects, as well as other objects which will 25 
become apparent from the discussion that follows, are 
achieved, according to the present invention, by replac- 
ing the synaptic weights between two processing ele- 
ments of the type shown in FIG. 2 with an adaptable- 
digital filter. Instead of a single synaptic weight (which 30 
with the standard back-propagation neural network 
represents the association between two individual pro- 
cessing elements), the invention provides a plurality of 
weights representing not only association, but also tem- 
poral dependencies. In this case, the synaptic weights 35 
are the coefficients to adaptable digital filters. 

The biological implication of this representation can 
be understood by considering that synapses undergo a 
refractory period — responding less readily to stimula- 
tion after a response. 40 

More particularly, the present invention is realized by 
providing a processing element (i) for use in a space- 
time neural network for processing both spacial and 
temporal data, the processing element being adapted to 
receive a sequence of inputs X(n), X(n — 1), X(n — 2) . . 45 
. , each input X(n) being comprised of K components 
xi(n), X 2 (n), . . . x/n),. . . xj/n). xj/n). The processing 
element comprises the combination of: 

(a) a plurality K of adaptable filters (F 1 /, F 2 /, . . . F )/, 

. . . Fj Ki) each filter F // having an input for receiving a 5Q 
respective component x/n), x/n— 1 ), x/n— 2 ), . . . , of 
the sequence of inputs, where x/n) is the most current 
input component, and providing a filter output y/n) in 
response to the input x/n) which is given by: 

yfn) -Aomjyfn - m), bkjxfn ~ *)). 

where a m j and by are coefficients of the filter F/V and 
f denotes the operation of the filter; and 

(b) a junction, coupled to each of the adaptive filters, 
providing a non-linear output p/(S/(n)) in response to the 60 
filter outputs y/n) which is given by: 

pKSKn))=Ayj(n)). 

In this case the junction presents a sequence of output 65 
signals, p/(S,{n)), p<S,<n-l)) f p,(S,<n-2)), 

At the network level, the invention is realized by 
providing a neural network for processing both spacial 


where a m j and b kj are coefficients of the filter F )/ and 
f denotes the action of the filter. 

Each first processing element (i) further comprises a 
first junction, coupled to each of the adaptive filters, 
providing a non-linear output p,{S/(n)) in response to the 
filter outputs y/n) which is given by: 

pX5X«»=Av/n))- 

In this case each first junction presents a sequence of 
first output signals, 

piSin% pKSKn-m ptftn-2)) 

The preferred embodiments of the present invention 
will now be described with reference to the accompa- 
nying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a representational diagram of a classical 
biological neuron. 

FIG. 2 is a block diagram of an artificial neuron or 
“processing element” in a back-propagation network. 

FIG. 3 is a block diagram showing the connection 
scheme for Jordan’s network architecture which learns 
to associate a static input with an output sequence. 

FIG. 4 is a block diagram showing the connection 
scheme for the Elman network wherein a history of the 
network’s most previous state is stored by transferring 
the activations in the hidden layer to a pseudo input, 
context layer. Longer term memories are attainable by 
adding recurrent connections to the context units. 

FIG. 5 a and FIG. Sb are representational diagrams of 
an S-plane and a Z-plane, respectively, illustrating the 
relationship between the continuous domain S-plane 
and the discrete domain Z-plane. 

FIG. 6a is a block diagram of a digital network for a 
finite impulse response (FIR) filter. 

FIG. 6b is a block diagram of a digital network for an 
infinite impulse response (HR) filter. 

FIG. 7 is a block diagram of a space-time processing 
element according to the present invention. 

FIG. 8 is a block diagram of a fully connected net- 
work utilizing space time processing elements accord- 
ing to the present invention. In this network, a set of 
input waveform sequences are mapped into an entirely 
different output waveform sequence. 

FIG. 9 is a graph of an error curve for the temporal 
XOR problem trained in a one input element, five hid- 
den element and one output element network with 5 
zeros and 0 poles between the input and hidden layers 


55 
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and 5 zeros and 0 poles between the hidden and output 
layers. 

FIG. 10 is a graph of an error curve for a two input 
element, eight hidden element and eight output element 
network with 5 zeros and 0 poles between the input and 5 
hidden layers and 5 zeros and 0 poles between the hid- 
den and output layers. 

FIG. 11 is a diagram showing the generation of a 
chaotic sequence by computer. 

FIG. 12 is a plot of a chaotic sequence generated by 10 
a process described hereinbelow. 

FIG. 13 is a diagram showing the space-time neural 
network’s performance on a chaotic problem after 900 
training passes. The ordinal numbers 525 through 625 
represent the network’s prediction. 15 

FIG. 14 is graph showing the testing performance 
(both maximum and RMS errors) as function of training 
passes. 

DESCRIPTION OF THE PREFERRED 2 0 
EMBODIMENTS 

Digital Filter Theory Review 

Before proceeding with a detailed description of the 
Space-Time Neural Network (“STNN”) system ac- 2 $ 
cording to the present invention, it is important to intro- 
duce digital filter theory and some nomenclature. 

Linear difference equations are the basis for the the- 
ory of digital filters. The general difference equation 
can be expressed as: 3 0 
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along the jo> axis. This is the well known Fourier 
transform. 

(3) A pole on the jo> axis, signifying no damping, 
produce a pure sinusoidal signal. However, a pole 
which travels onto the left half plane of the s-plane 
exponentially increases, eventually sending the 
system into an unstable state. 

The discretized form of the Laplace transform has 
been developed further and is referred to as the z-trans- 
form. The notation z~i is used to denote a delay equal 
to one sampling period. In the s-domain, a delay of T 
seconds corresponds to e~ sT . Therefore, the two vari- 
ables s and z are related by: 

z~'=e-* T ( 8 ) 

where T is the sampling period. The mapping between 
the variables can be further illustrated by referring to 
FIG. 5. First notice that the left half plane of the s-plane 
maps to the area inside a unit circle on the z-plane. In 
abiding with the Nyquist criterion, sampling at least 
twice the signal bandwidth, fy note that as one traverses 
from — f/2 to + f/2 on the s-plane, it is equivalent to 
going from n radians toward 0 radians and back to 7 r 
radians in a counterclockwise direction on the z-plane. 
Furthermore, note that lines in the s-plane map to spi- 
rals in the z-plane. 

By evaluating the z-transform on both sides of the 
linear difference equation, it can be shown that 


N M 

yin) = 2 bkx(n — k) + 2 - m) 

k—0 m— 1 


( 6 ) 


where the x and y sequences are the input and output of 35 
the filter and a m ’s and b^-’s are the coefficients of the 
filter. Sometimes referred to as an s-transform, the well 
known continuous domain Laplace transform is an ex- 
tremely powerful tool in control system design because 
of its capability to model any combination of direct 40 
current (DC) or static levels, exponential, or sinusoidal 
signals and to express those functions algebraically. The 
s-plane is divided into a damping component (cr) and a 
harmonic component (jco) and can mathematically be 
expressed as 45 

s ^ € -(a+M (7) 

This formulation has a number of interesting character- 
istics as follows: 50 

(1) The general Laplace transfer function can be 
thought of as a rubber sheet on the s-plane. A desir- 
able transfer function is molded by strategetically 
placing a transfer function’s roots of the numerator 
and the denominator in their appropriate positions. 55 
In this case, polynomial roots of the numerator are 
referred to as zeros and “pin” the rubber sheet to 
the s-plane’s ground. On the other hand, polyno- 
mial roots of the denominator are referred to as 
poles and their locations push the rubber sheet 60 
upwards— much like the poles which hold up the 
tarpaulin in a circus tent. Therefore, zeros null out 
certain undesirable frequencies and poles can either 
generate harmonic frequencies (if close enough to 
the jco axis) or allow certain frequencies to pass 65 
through the filter. 

(2) Setting the damping coefficient, cr , to zero is ef- 
fectively similar to taking a cross sectional cut 


Rrt-JSL. 
nz) ~ X(z) 


N i 
2 b k z~ k 

k~ 0 

M 

1 - 2 a m z~' 

nt= 1 


(9) 


Digital filters are classified into recursive and non- 
recursive types. Filters of the nonrecursive type have 
no feedback or recurrent paths and as such all the a m 
terms are zero. Furthermore, digital filters are also clas- 
sified in terms of their impulse responses. Because non- 
recursive filters produce a finite number of responses 
from a single impulse, such filters are referred to as 
“Finite Impulse Response” (“FIR”) filters. On the other 
hand, the recursive filters produce an infinite number of 
responses from an impulse and are therefore referred to 
as “Infinite Impulse Response” (“HR”) filters. For ex- 
ample, if a unit impulse is clocked through the filter 
shown in FIG. 6 (a), the sequence 

bo, bi, b 2 , . . . b H , O, O, 0,0,0,... O, O, O 

will be the output. Notice that the filter produces only 
the coefficients to the filter followed by zeroes. How- 
ever, if a unit impulse is presented to the filter shown in 
FIG. 6(6), because of the recursive structure, the re- 
sponse is infinite in duration, 

FIR and IIR filters each possess unique characteris- 
tics which make one more desirable than the other de- 
pending upon the application. The most notable of these 
characteristics include: 

(1) FIR filters, because of their finite duration are not 
realizable in the analog domain. IIR filters, on the 
other hand, have directly corresponding compo- 
nents in the analog world such as resistors, capaci- 
tors, and inductive circuits. 

(2) IIR filters cannot be designed to have exact linear 
phase, whereas FIR filters have this property. 
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(3) Because of their recursive elements, HR filters are 
an order of magnitude more efficient in realizing 
sharp cutoff filters than FIR filters. 

(4) Because of their nonrecursiveness, FIR filters are 
guaranteed to be stable. This property makes FIR 5 
filters much easier to design than HR filters. 

These different properties between FIR and HR fil- 
ters must be carefully weighed in selecting the appropri- 
ate filter for a particular application. 

DESCRIPTION OF THE SPACE-TIME NEURAL 10 
NETWORK 

Having introduced digital filter theory, it is now 
possible to proceed with the detailed description of the 
Space-Time Neural Network (STNN) system accord- 15 
ing to the present invention. What follows is a detailed 
procedure for constructing and training the STNN. As 
mentioned earlier, in the STNN system the weights in 
the standard back-propagation algorithm are replaced 
with adaptable digital filters. The procedure involves 20 
the presentation of a temporal ordered set of pairs of 
input and output vectors. A network must consist of at 
least two layers of adaptable digital filters buffered by 
summing junctions which accumulate the contributions 
from the subsequent layer. 25 

A pictorial representation of the space-time process- 
ing element is illustrated in FIG. 7. In this case, a value, 
say x/n), is clocked in to its associated filter, say Fy,<n), 
producing a response y/n) according to the filter repre- 
sentation 30 


yfa) = 


M 

2 flm/T/n - m) + 
tn~\ 


N 

1 

k = 0 


bkjx/n - k) 


( 10 ) 


All remaining inputs are also clocked in and accumu- 
lated by the summing junction i: 


Sfa) - 2 jfjn) 
all j 



The contributions from the signals fanning in to the 
summing junction are then non-linearly transformed by 
the sigmoid transfer function 

45 


Pi(SKn)) - 


1 

1 -j- e — $/(«) 


( 12 ) 


This output is then made available to all filter ele- 
ments connected downstream. 50 

As explained earlier, the space-time neural network is 
comprised of at least two layers of filter elements fully 
interconnected and buffered by sigmoid transfer nodes 
at the intermediate and output layers. A sigmoid trans- 
fer function is not used at the input. Forward propaga- 55 
tion involves presenting a separate sequence-dependent 
vector to each input, propagating those signals through- 
out the intermediate layers as was described earlier until 
reaching the output processing elements. In adjusting 
the weighing structure to minimize the error for static 60 
networks, such as the standard back-propagation, the 
solution is straightforward. However, adjusting the 
weighing structure in a recurrent network is more com- 
plex because not only must present contributions be 
accounted for but contributions from past history must 65 
also be considered. Therefore, the problem is that of 
specifying the appropriate error signal at each time and 
thereby the appropriate weight adjustment of each co- 


10 

efficient governing past histories to influence the pres- 
ent set of responses. 

The procedure for adjusting the weighing structure 
for the space time network is as follows: 

First compute the errors at the output layer for each 
processing element, i, using the formula: 

6 i=mk)-Am)r(m)) 03 ) 

where: 

D,<k) is the kth desired response from a given se- 
quence for neuron i at the output layer 
A/(k) is the network’s output response at neuron i for 
the kth input sequence pattern 
P'(E/(k)) is the first derivative of the sigmoid function 
for the ith output’s activation value or 
P(E,<k))(l-P(E<k)) 

Next to compute the updates for the coefficients of 
each filter element between the hidden and output layer 
processing elements, a reversal procedure is imple- 
mented. Whereas in the forward propagation, input 
values were clocked into the filter elements, back- 
propagation instead involves the injection of errors into 
the filter elements according to the formula: 


A6/yJt0H- l)=a[i]A|b, ;*(/»)+ (I -17)8^;] 


where: 

Aby^n + l) is the update for a zero coefficient, b*, 
lying between processing elements i and j 
a is the learning rate 

Abijkin) is the most recent update for the kth zero 
element between processing elements i and j 
7] damps most recent updates 
8/ is described by (13) 

X// contain a history of the output of the jth neuron in 
the hidden layer 

The recursive components in each filter element are 
treated the same way and are updated according to the 
formula: 

Aa,]k(n+i)=a[7}Aa l jk(n)+(l-7i)6/Yyt c ] (15) 

where: 

Aaijk(n+ 1) is the update for a pole coefficient, a*, 
lying between processing elements i and j 
a is the learning rate 

Aa//*(n) is the most recent update for the kth zero 
element between processing elements i and j 
7} damps most recent updates 
6/ is described by (13) 

Y ijk contain a history of the activation values for the 
non-recursive filter elements between neurons i 
and j, k time steps ago 

For implementation purposes, the present algorithm 
only considers the accumulation of errors which span 
the length of the number of zeroes between the hidden 
and output processing elements. Thus: 


S/jfc =- 2 F{Eik)hfiijk 


(16) 


where: 

i is the index of the hidden neuron 
j ranges over the neuron idices for the output layer 
8j is described by (13) 
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P'(E/*) is the first derivative of the sigmoid function 
for the kth history of activation levels for the ith 
neuron in the hidden layer 
6 fiijk sums the results of injecting the previously com- 
puted errors found in equation (13) through the 
FIR portion of the filter element, by*, found be- 
tween the ith neuron in the hidden layer and the jth 
neuron in the output layer. 

Simulations 
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The space-time neural network according to the pres- 
ent invention was constructed and tested to perform a 
number of simulations. Source code for a computer 
program written in “C” language for simulation of the 
STNN is included in the Appendix below. 

The first simulation test was a variation of the classic 
XOR problem. The XOR is of interest because it cannot 
be computed by a simple two-layer network. Ordinar- 
ily, the XOR problem is presented as a two bit input 
combination of (00, 01, 10, 1 1) producing the output (0, 
1 1, 0). 

This problem was converted into the temporal do- 
main in the following way. The first bit in a sequence 
was XOR’d with the second bit to produce the second 
bit in an output sequence; the second bit was XOR’d 
with the third bit to produce the third bit in an output 
sequence, and so on, giving the following: 


Input 1 0 1 0 1 0 0 0 0 1 1 

Output 0 1 1 1 1 1 0 0 0 1 0 


In the simulation, the training data consisted of 100 
randomly generated inputs and the outputs were con- 
structed in the manner described above. A network was 
implemented which had 1 input element, 5 hidden ele- 
ments and 1 output element and had 5 zero coefficients 
and 0 pole coefficients between the input and hidden 
layers and 5 zero coefficients and 0 pole coefficients 
between the hidden and output layers. The task of the 
network was to determine the appropriate output based 
on the input stream. The error curve for the network 
showing the RMS error in dependence upon the num- 
ber of training passes is shown in FIG. 9. 

For a second simulation, a network was implemented 
with 2 input elements, 8 hidden elements and 8 output 
elements having 5 zeros and 0 poles between input and 
hidden, and 5 zeros and 0 poles between hidden and 
output layers. A problem, called the Time Dependent 
Associative Memory Test, was constructed which 
would test the network’s ability to remember the num- 
ber of events since the last trigger pattern was pres- 
ented. The data consisted of 1000 input/output pairs 
where the input bits were randomly constructed and the 
output appropriately constructed. As an example, con- 
sider the first 7 sets of data in the following list. Note 
that a “1” bit sequentially gets added to the output for 
the input patterns 0 0, 1 0, 1 0, 0 0, 1 0, and 0 1 until the 
1 1 pattern is presented which resets the output back to 
the 1 0 0 0 0 0 0 0 state. 


Input 

0 0 

1 0 

1 0 

0 0 

1 0 

0 1 


1 1 
1 1 
1 1 
1 1 
1 1 
1 1 


Output 

0 0 0 0 0 

1 0 0 0 0 

110 0 0 

110 0 0 

11110 
1 I 1 1 1 


0 

0 

0 

0 

0 

0 


The error curve for this Time Dependent Associative 
Memory Test is shown in FIG. 10. As will be seen from 
FIGS. 9 and 10, the RMS error converged rapidly 
toward zero. 

D The final simulation illustrates that the space-time 
neural network according to the present invention is 
able to learn the dynamics and model the behavior of a 
chaotic system. The graph shown in FIG. 11 is a plot of 
a sine function extending from 0 to v with amplitude n. 
20 A “chaotic sequence” can be generated by randomly 
selecting a value between 0 and t r, say xo, determining 
the value of 7r-sine(xo) to produce xj, and repeating this 
iterative process into a general form represented by 
X n _}-i=7r-sine (x„). FIG. 12 shows a collection of x’s 
25 generated by this process. 

The goal of the STNN system in this simulation was 
to predict a future point, given a history of past points. 
To keep within the linear bounds of the sigmoid, the 
sequences collected above were normalized such that 
30 the range from 0 to n mapped into the range from 0.2 to 
0.8. An STNN system was constructed with 1 input 
element, 6 hidden elements and 1 output element, with 
10 zeros and 0 poles between the input and hidden lay- 
ers, and 10 zeros and 0 poles between hidden and output 
35 layers. The system was trained with 525 data points. 
Training was periodically suspended to test the system 
by stimulating it with the sequence of the last 50 samples 
of the training set — the ordinal values 475 to 525. At this 
point, the system was prepared to make its first predic- 
40 tion. The predicted value could have been fed back into 
the input to generate a new predicted value. Instead, the 
system was fed with actual values generated by the 
chaos algorithm — that is, ordinal numbers 526 through 
625. FIG. 13 illustrates the system’s performance at 
45 various stages during the training process. FIG. 14 
shows the average error of the network’s performance 
during the training process. 


50 Output 0.31 0.51 0.80 0.21 0.22 0.25 0.37 . . . 

Input 0.51 0.80 0.21 0.22 0.25 0.37 0.67 0.59 


Conclusion 

55 The space time neural network (STNN) is a general- 
ization of the back-error propagation network to the 
time domain. By adopting concepts from digital filter- 
ing theory, the network is afforded a distributed tem- 
poral memory which permits modeling complex dy- 
60 namic systems and recognition of temporal sequences as 
shown in the simulations. The STNN architecture dif- 
fers from previous work of Jordan and Elman in that 
the network’s memory is distributed over the connec- 
tions rather than being implemented as a special layer of 
65 nodes. This distinction allows the STNN to possess an 
adaptive temporal memory without introducing addi- 
tional nonlinearities into the learning law; i.e., the action 
which occurs on connections between nodes is still 
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linear, while the nonlinear actions occur within the 
nodes. 

There has thus been shown and described a novel 
space-time neural network for processing both spacial 5 
and temporal data which fulfills all the objects and 
advantages sought therefor. Many changes, modifica- 
tions, variations and other uses and applications of the 


14 

subject invention will, however, become apparent to 
those skilled in the art after considering this specifica- 
tion and the accompanying drawings which disclose the 
preferred embodiments thereof. All such changes, mod- 
ifications, variations and other uses and applications 
which do not depart from the spirit and scope of the 
invention are deemed to be covered by the invention, 
which is to be limited only by the claims which follow. 


U.S. Gov’t 

* MSC-21874-1 


APPENDIX 


Patent Application 


/* Hit: itnn_con.h— portability and coomon declarations for tha Space •/ 

/• Tine Neural Network code. •/ 

/* by A. 0. Shelton and J. A. Villarreal */ 

/• A product of the Software Technology branch of NASA/JSC */ 

/* Any duplication or distribution of this code without the express consent •/ 
/• of NASA is a violation of Federal Law. »/ 

•include <stdio.h> 

•include <string.h> 

•Include <aath.h> 

•define TbC 0 
•define true *\01' 

•define false '\0' 

•define Stability_Threshold 1000.0 
•define machine zero 0.00001 
•define min U, bT (Ha)<{b)>? (a): (b) ) 

•define square(x) (<x)*(x)) 

• define sigmoid (x) (1.0/ (1.0+exp (- (x) ) ) ) 

• define d sigmoid(y) ( (y) * (1.0- (y) ) ) 

•if TBC 

•Include <alloc.h> 

•define r_b 15 
•define GIGANTIC huge 
• •else 

•Include <malloc.h> 

•define r b 15 
•define GIGANTIC 
•endif 

• define getchO (gets (strl) [0] ) 

• define frand(x,y) { (x) ♦ ( (y) - (xj ) • ( (rand () a ( (ll«r b) -1) ) /\ 

(float) (ll«r_b)}) 

typedef char string (256); 


FILE •HidFile; 

int nO • 1, nl • 6, n2 - 1, nzO ■ 5, 

npO • 0, NuaSamples, NuaTestSamples, NumSets, NumTestSets, npl - 0, nzl - 5; 

long total_cycles - 0; 

float alpha • 0.2, delta • 0.9, epsilon; 

float "lvO, ** # ov0, •••IcO, ***oc0, •••dicO ,** # dcc0, •••aicO, •••rnocO, 
•••ihO, ••**oh0, 

••ivl, * * *ov 1 , •••icl, •••©€!, •••did, •••docl, •••micl, •••mod, 

•nvO, *nvl, 

•OutError, •HidError, ••Waveln, **WaveOut, **TestWaveIn, ••TestWaveOut; 

string strl, str2; 

•Include <stdlo.h> 

•Include ^strings. h> 

typedef char string(2Sf); 

extern string strl, str2; 

extern int nO, nl, n2, — 
nzO, npO, nzl, npl; 
extern float # **ic0, •••oc0, 

•••id, •••ocl? 

/* get coefficients from file name s */ 
void get_coef ficients (s) 
char *s;~ 

( 

. int i, j; 

FILE *f ; 
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15 

while ((Mep«n(i, "rb")) -• HULL) 
i 

printf (• {111 %* not found\n fiienane> ■, *); 
gets(s) ; 

) /• end while •/ 
for (i - 0; 1 < nX; !♦♦) 

for (3 • 0; j < nO? 3*+> 
i 

If (nzO > 0) 

freed ((ch*r*)ic0(3) (1), 

(unsigned) sixeof (float) , nxO, f); 

If (npO > 0} 

freed ( (chsr*)oc0(3) (i) , 

(unsigned) sixeof (float) , npO f f); 

) /• end for 13 */ 
for (i • 0; i < n2; i**) 

for (3 - 0; 3 < nX; 3^) 

( 

if (nxl > 0) 

freed ( (char*) icl (31(1) « 

(unsigned) sixeof (float) , nxl, f); 

If (npl > 0} 

freed { (ehar*)ocl (31 (1), 

(unsigned) sixeof (float) , npl, f); 

J /* end for 13 */ 
fclose (f); 

} /* end get_coefficients */ 

/• save coefficients in file name s •/ 
void save_coefficients (a) 
char *s; 

I 

int i, 3# ret; 

TILE *f, *fl; 

/* if (fopen (s, *r“) i- HULL) 

( 

printf (• file %s existsSn new nane or carriage return to overvrite> ", s); 
if (strlen(gets(str2) }) 
strepy (s, str2); 
fclose (f); 

1 end if •/ 
f - fopen (s, "wb-) ; 
for (i • 0; i < nl; i**) 

for (3 • 0; 3 < nO; 3*e) 

i 

if (nzO > 0) 

fwrite ( (char*)4cO(3) (1), 

(unsigned) sixeof (float) , nzO, f); 
if (npO > 0) 

fwrito ( (char*)oc0f 3! [11, 

(unsigned) si zeof (float) , npO, f ) ; 

) /• end for i 3 •/ 
for (i • 0; i < n2; !♦♦) 

for (3 • 0? 3 < nl; 3*+) 

( 

if (nxl > 0) 

fwrite ( (char*) icl 1 3 1 (1 ) • 

(unsigned) sixeof (float) , nxl, f); 
if (npl > 0) 

fwrite ( (char*)ocl( j) (1), 

(unsigned) sixeof (float) , npl, f); 

) /* end for 13 */ 

fclose (f); 

} /• end save_coeff icients */ 

•include "stnn^eom.h" 

/* general memory allocation routine */ 


char *mem_alloe (n) 
int n; 
l 

char *r; 

if (n > 0) 

{ 

r •» (char*) malloc (n); 
if (r •• HULL) 

I 

printf ( M me* alloc: We are out of memory !\n“) ; 

exit (0); 

) /* end if */ 
return r; 

) /• end if */ 
else 
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rtturn HULL; 

) /* and ita^illoc •/ 

/* Declares a three dimensional array of slat L X N x N. Returns th« 
address to the three dimensional array V 

float*** DeclareTripleArray (L, M, H) 
lnt L, H, N; 
l 

int 1, j; 
float ***Trlple; 

Triple • (float***)»em_alloc(L • siseof (float**)}; 
for (1 • 0; i < L; 1++7 
( 

Tripled) - ( float **)maa_allec(M • aixeof (float*) J; 
for (3 • 0; 3 < K; j++) 

Tripled) (3) • (float *)aea alloc (K • tlxcof (float) ) ; 
I /• end 1 •/ *" 

return (Triple); 

) /* end DeclareTripleArray */ 


f 


/* 

Declares a two dimensional array of site LXN . Returns the 
address to the two dimensional array 
•/ 


float** DeclareDoubleArray (L, M) 
int L, M? 

( 

int 1 ; 

float **Double; 

Double • (float **)mam_a Hoc (L * siseof (float*)); 
for (i • 0; i < L; 

Doubled) - (float *)mem_a lloe (H * slzeof (float) ) ; 
return (Double) ; 

) /• end DeclareDoubleArray */ 

void reset network () 

( 

int 1 , 3 , k; 

for (1 • 0; 1 < n0; !♦♦) 

( 

for (k - 0; k < nsO? k**) 

( 

ivOdl (kj • 0 . 0 ; 

for <3 - 0; 3 < nl? 3 *+) 

dlc 0 (ll ( 3 ) (k) - *ic 0 dl( 3 Mkl - 0 . 0 ; 

| /* end for k */ 

for (3 - 0 ; 3 < nl; 3**> 

l 

for (k • 0; k < npO; k**) 

ov0(iM3)Ik| - doc0(il(3) (k| - moc 0 (i} (31 (kj - 0 . 0 ; 
ovO (i) (3) (npO) - 0.0; 

J /* end for 3 •/ 

! /* end for i */ 

for (i - 0 ; i < nl; 1 ++) 

( 

for (k - 0 ; k < nzl; k++) 

( 

ivld) (k) - 0.0; 

for (3 - 0 ; j < n 2 ; y+) 

dlcl(i) ( 3 ) (k) - micl(i) ( 3 ) (k) - 0 . 0 ; 

I /• end for k */ 
for (3 - 0 ; 3 < n 2 ; 3*+> 

{ 

for (k - 0 ; k < npl; k*+) 

ovldl (31 (k) - docldJ(31tk! -mocidldllkl -0.0; 
ovl d) (31 (npl) - 0 . 0 ; 

) /* end for 3 */ 

} /* end for i •/ 

) /* end reset_network */ 

void allocate network () 
t 

int 1 , j, k; 
float wtl&O, wtl&l; 

prlntf (* enter sizes of input hidden and output layers <%d %d %d> “,n 0 ,nl,n 2 ) 
sscanf (gets (strl) # *%d%d%d",*nQ,*nl, 4 n 2 ) ; 

prlntf (* enter numbers of zeros and poles from input to hidden <td %d>*, 
nzQ.npO) ; 
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sscanf (gets (*trl) , Md%d", into, snpO) ; 
printf (" antar nuabars of taros and polos fro* hiddan to output <%d %d> •, 
nil, npl); 

sscanf (goes (strl) , Md*d*, *nxl, 4 npl) ; 

/• incraaant nx's bocauso must hava at loast 1 Input coafficiant */ 

nt0+*; 

nxl**; 

nvO - (float* jBon^alloe (nl*sliaof (float) ) ; 
nvl - (float*)mam~alloc(n2*slzaof (float) ) ; 

KidCrror • (float*) »am_alloc (nl*sltaof (float) ) ; 

Out Error • (float*) mam~alloc(n2*slxaof (float) ) ; 

lvO • Do c la ra Do ub la Array (nO, nsO); 

lvi * Dacia raDoub la Array (nl,nzl) ; 

ovO • Dacia raTrlplaArray (n0,nl, np0*l) ; 

IcO - DaciaroTrlpiaArray (nO, nl, ntO) ; 
dlcO • DaclaraTrlploArray (nO, M. ntO) ; 
ocO • Dae la raTrlplaArray (n0,nl, npO) ; 
docO - Dacia raTrlplaArray (nO, nl.npO) ; 
alcO • DaclaraTrlplaArray (nO, nl, ntO) ; 

■ocO • DaclaraTrlplaArray (nO, nl,npO> ; 
ovl - DaclaraTrlplaArray (nl,n2, npl*l) ; 
lei - OaclaraTrlplaArray (nl,n2, ntl) ; 
did • DaclaraTrlplaArray (nl,n2, ntl) ; 
ocl * DaclaraTrlplaArray (nl,n2, npl) ; 
docl - DaclaraTrlplaArray (nl,n2, npl) ; 

■icl - DaclaraTriplaArray(nl,n2,ntl) ; 
aocl - DaclaraTrlplaArray (nl,n2,npl) ? 

IhO * (float***)maa_alloc(nxl*tixaof (float**)) ; 
ohO - (float •***)ma« alloc (nxl*sixaof (float***) ) ; 

Cor U - o; 1 < nil?“too) 
l 

ihO(U - DaclaraDoublaArray (nO, nxO) ; 
ohO (ij • DaclaraTrlplaArray (nO, nl f npO+1) ; 

) /* and for 1 •/ 
printf {“ saad> •) ; 
if (strlan (gats (strl) >> 

srand (atoi (strl) ) ; 

also 

srand (clock () ) ; 

wtlmO • *in (0.5, 2.0/sqrt ( (float ) nO) ) ; 
wtlml • min{0.5, 2,0/sqrt ( (float) nl) ) ; 
for (1 • 0; i < nO; i++) 

for (3 - 0; j < nl; 3-n-} 

( 

for (k • 1? k < nsO; k+o) 

icO(i) (31 (k) - 0.0; 

IcO f 1 ) I3J (0) - frand (-wtlmO, wtlmO); 
for (k • 0; k < npO; k*<M 

ocO(i] (31 Ck) - 0.0; 

} /• and for 3 •/ 
for (1 - 0; 1 < nl; !♦♦) 

for (3 • 0; 3 < n2; 3++) 

( 

for (k • 1; k < ntl; k* + ) 

icl(l)(3J(k) - 0.0; 
icl (11 (31 (01 - frand (-wtlml, wtlml); 
for (k - 0; k < npl; k++) 

ocl (i 1 (31 (k J - 0.0; 

1 /• and for 3 */ 

) /• and allocata_nacwork •/ 

/• This routina gats tha I/O training data. "Wsvaln* and ‘WavaOut* 
ara two dlmansional arrays. "NavaXn* has tha dlnansions 
nO X NumSamplas and -WavaOut* has tha dimansions 
n2 X KumSamplas. nO and n2 should alraady ba 
daclarad on Input. 

•/ 

void gat lo () 

( 

int i, HI, NO; 
float ul; 

FIU *f, *fl; 

printf (* Natwork Canarallxatlon Tast flla nama> •) ? 
whlla ( (fWopan(streat (gats (strl) , -.tsf) , "r") ) ••NULL) 

printf (• .tst fila \» not found\n fila nana> *, strl); 

If (fscanf(fl, "tdld", (NumTastSamplas, (NuaTastSats) !• 2) 

( 

printf (* no haadar lina fila found In Tast rilaiVn*}; 
a*it (0); 

) 

printf (* l/o fila nama> "); 

whila ( (Wopan (s treat (gats (strl) , ■ .lop") , *r*J ) *-NULL) 

printf (• . iop fila %s not found\n fila na*a> -, strl); 
if (fscanf (f, *%dldld*d\ SNI, SNO, tNumSamplas, SNumSats) !- 4) 
i 

printf (* no haadar lina in l/o fila!\n"); 



21 


5,253,329 


22 


•XU (0) ; 

I /* and if •/ 

if ( (HI !«n0) 1 1 (NO!-n2) J 

i 

print f ( * i/o fil« dots not watch natvork spacification!\n*l; 
•xit <0); 

1 /• ond if */ 

Wavaln * DaclaraDoubltArray <n0 # (NumSats * NumSamplas) ) ; 

WavaOut • DaclaraDoublaArray (n2, (NumSats • NumSamplas) ) ; 

TastWavaln • DaclaraDoubltArray (nO, (NumTastSats • NmiTastSamplas) ) ; 
TastWavaOut • DteiaraDoublaArray (n2r (NumTastSats • NumTastSamplts) ) ; 
for (i • 0; i < (NumTastSats • NumTastSamplas) ; i**) 

{ 

for (HI • 0; MX < n0; HI**} 

I 

if ffscanf (fl, Mf, iul) ~ 0) 

( 

print f (• incomplata last filt!\n*}; 
axit (0); 

} /* and if */ 

TastWavtIn(NIl {i| - ul; 

} /* and NX */ 

for (HO • 0? HO < n2; NO**) 

( 

if (fseanf (fl t -%f. «ui> — 0) 

C 

print f (• incomplata Ttst fila!\n*); 
axit (0); 

} /* and if */ 

TastWavaOut (NO) (i] * ul; 

) /* and HO •/ 

I 

for (i • 0; i < (HuaSaaplas • NumSats); i++) 

{ 

for (HI • 0; HI < nC; NI**) 

( 

if (fscanf (f, -%f, Sul) — 0) 

( 

printf (* incomplata i/o fila!\n p l; 
axit (0); 

) /• and if •/ 

WavtIn(NI! (11 - ul; 

) /• and HI */ 

for (HO - 0; HO < n2; NO**) 

i 

if (fscanf(f, ■If, 4ul) •• 0) 

{ 

printf (" incomplata i/o fila!\n">; 
axit (0); 

) /• and if */ 

VavaOut (NO] [ 1] - ul; 

1 /* and HO •/ 

\ /* and for i •/ 
fclosa <fl; 

} /• and gat_io •/ 

void print arr 0 
i 

int ir *siza; 
float *t; 

printf (* hiddan or output arror <h/o> •) ; 
if (gatchO — 'h*) 

( 

siza - «nl; 
t - Hidtrror; 

) /* and if •/ 

alsa 

( 

siza * «n2; 
t - OutError; 

} /♦ and alsa •/ 

for (i • 0; i < *fizt; i**) 

( 

printf (• *6.2f, t(iU? 
if ((i%6) — 5) 

putchar {'\n')? 

) /• and for i •/ 
if (! U%6) 1 

putchar ('\n'); 

) /• and print_arr */ 
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void print correction O 
i 

int c - 0 # i, j # k, •sixe_i, *sixejc; 

float ***t; 

prlntf (* proas return for helpSn"); 

start: 

printf (• cholce> *); 
switch (atoMgets (strl) )) 

I 

case {1} : 

•ixe_i • inO; 

sixe~3 • snl; 

sixe~k - tnxO ; 
t • dieO; 
break; 

} /* end 1 •/ 
case (2): 
l 

sixe_l * snO; 
sixe”3 • ini; 
sixejt • inpO; 
t - docO; 
break; 

I /• end 2 •/ 
case (3): 

( 

sixe_l • ini; 
si re” 3 • in2; 
sixe~k • inxl; 
t - did; 
break; 

I /• end 3 •/ 
case (4): 

[ 

sixe_i • ijil; 
sixe~3 • in2; 
sixejt • inpl; 
t • docl; 
break; 

I /• end 4 •/ 

default : 

( 

printf (*choices:\n“) ; 

printf (•!: display input to hidden D^Input_Coeff iclent\n*) ; 

printf (*2: display input to hidden D_Output_Coefficient\n*) ; 

printf i*3: display hidden to output 5jnput~Coef ficlentVn") ; 

printf {"4: display hidden to output D^Output^CoeffieientVn*) ; 

goto start; 

) /* end default */ 

| /* end switch V 
for (1 • 0; i < *sixe i; 

! 

printf (• fro* node %d\n*,i); 
for (j - 0; j < •sixej; j++) 

1 

printf (" to node %d\n", j); 
for (k • 0; k < *slxe k; k++) 
i 

printf r I6.2f* r t[l]{))[k)); 
if ( « 5) 

putchar ('\nM; 

} /* end for k */ 

) /• end for 3 */ 

J /* end for i */ 
if <! (c%i)J 

putchar <'\n'); 

) /• end print_correction */ 

void print weights 0 
( 

int c-0, i, 3, k, slte_l, sixe_3, sixejcl, slxejt2; 
float * # *tl, ••*t2; 

printf (• hidden or output weights <h/o> •) ; 
if (getchO — 'h') 

I 

sixe_i • n0; 
sixe_3 • nl; 
slxejtl - nxO; 
sil«”k2 - npC; 
tl *”ic0; 
t2 • ocO; 

} /• end if •/ 

else 

i 
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sixe_l - nl; 
size~J - n2; 
slzejtl • nxl; 
sixe“k2 - npl; 
tl -~icl; 
t 2 • eel; 

J /• end else */ 
for (1 • 0; 1 < size i? H*) 
t 

printf (• fro« node %d\n“, i); 
for (j • 0; J < sixej; j++) 
l 

printf (• to node 4d\n*, J); 
printf C Input coeff lclents:\n") ; 
for <k • 0; k < slzejtl; 

printf (• %7.4f-, tllUUlfkl)? 

If ( (c**%6) ••5) 

putchar ( # \n'); 

I /• end for k •/ 

If (2 <c%«n 

putchar ( # \n # ); 

printf (• output coefflclents:\n*); 
for (k • 0; k < site k2; k^) 
t 

printf (- I7.4f, tJUUJIUll; 
if (<c++%«)~5) 

putchar t'\n'); 

I /* end for k */ 
if (!(€«<)) 

putchar t'\n'j; 

I /♦ end for j •/ 

) /* end for 1 •/ 

| /* end print_weights */ 

/* compute output of filter in response to Input x */ 

/• maintain input_value and outputjulue arrays which contain */ 

/• respectively histories of Inputs and outputs •/ 

/* starting with the most recent. */ 

/* compute and accumulate gradient descent vectors d_lnput_eoefflcients •/ 

/• and d_output_coeff ieient a for the coefficient arrays •/" 

/* the Input parameter is the amount of error to be fed back •/ 
void gradient (d_input_coefficient, input_value, 
d_out put_coeff ieient, out put jralue, NuaZeros, NuaPoles, dy) 
float *d~input_coef f Ieient, ~input_value, 

•d_output_coeff ieient, *output_value, dy; 
int NuaZeros, NuaPoles; ~ 

{ 

int i; 

for (1*0; 1 < NuaZeros; !«•♦) 

d^lnput^coefficient [i] ♦- 
(dy*input_yalue (i ) ) ; 
for (1 • 0; 1 < Nv» Poles; 1++) 

d_output_coeffieient(i) ♦- 
<dy*output_value(i*l] ) ; 

J /* end gradient •/ 

/* correct coefficient vectors from the •/ 

/• descent vectors •/ 

void apply_eorrectlon (InputCoefficient, D_InputCoef f ieient, 

OutputCoefficient, D_OutputCoeff ieient, a_T, a_o, FroaNode, ToNode, 

NuaZeros, NuaPoles) ~ “ 

float ***InputCoeff ieient, ***DJ(nputCoeff ieient, 

•••CutputCoef ficient, •••D^Out put Coeff ieient, ***a_i, •••o - o; 
int FroaNode, ToNode, NuaZeros, NuaPoles; ” “ 

{ 

int i; 

for (1 - 0; 1 < Hum Zeros; 

{ 

InputCoeff ieient (FroaNode 1 (ToNode) (i) *• 

(alpha* (epailon*D_InputCoef ficient (FroaNode) (ToNode) (i)*delta*a_l 
(FroaNode) [ToNode J (i) )) ; 

m_i (FroaNode) (ToNode) (1) - D_InputCoeff ieient (FroaNode) (ToNode) (i); 
D~InputCoef ficient [FroaNode lfloNode ) (1) » 0.0; 

) /• end for i •/ 

for (1 - 0; i < HuaPoles; i*+) 

( 

OutputCoefficient (FroaNode) (ToNode) (1) ♦- 

(alpha* (epsilon*D OutputCoefficient (FroaNode] (ToNode) [l)+delta*a_ 
o (FroaNode) (ToNode) (i)>); 

a_o( FroaNode) (ToNode) (i) - D_OutputCoef f ieient [FroaNode ) (ToNode) ( i I ; 
D~0utputCoef f ieient (FroaNode) (ToNode) (i J - 0.0; 

J /• end for i */ 
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void Props gate Forward (Staple, low, high, NetXnput) 
float low, high, **MetInput; 
lnt Sample; 

( 

lnt 1, In, Hid, Out? 
float t; 

float **iptr, ***optr; 


); 


ID) 


/* clear the hidden layer end output neurons •/ 

for (Hid - 0; Hid < nl; Hid+*j ^ 

nvO(Hld) * 0.0? 

for (Out • 0? Out < n2; Out**} 
nvl (Out | • 0.0? 

/* propagate input to hidden for Input staple "Sample* •/ 
for (In • 0? In < nO? In**} 

( 

for (1 • nx0-l; 1 > 0? 1—) 

ivOtlnldl - lv0(lnl (t-U ? 
iv0(ln)(0} - Xetlnput (In) (Sample j *frand (low, high) ; 
for (Kid • 0? Kid < nl? Kid**) 

{ 

t • 0.0? 

for (1 - 0? 1 < nxO; i**) 

t*-(lvO(In] f i | • IcO ( In ) (Hid] (!])? 
for (1 - npO? 1 > 0? 1— ) 

t*- (ocO (In] (Kid) (1-1] * (ovO(In) (Kid] (i]-ov0(ln| (Hid) (1-1} > 

nv0(Hid)*-t? 

ovO (In] (Hid) (0) « t? 

I /• end for Hid •/ 

} /• end for In */ 

for (Hid - 0? Kid < nl? Hid**) 

nv0(Hld] • sigmoid(nvO(HidJ ) ? 
if ( Hidrile !- HULL} 

(for (Hid • 0? Hid < nl? Hid**) 
fprintf (Hidrile, • %8.4f\t*, nv0(Hld])? 
f print! (HidFile, *\n*}; 

} 

/• propagate hidden to output */ 
for (Hid • 0? Hid < nl? Hid**) 

( 

for (i - nx 1-1; 1 > 0? 1— ) 

Ivl (Hid ](i] • Ivl (Hid) (1-1) ? 
ivl(Hld) (0] - nvO (Hid] ? 
for (Out • 0? Out < n2? Out**) 

( 

t • 0? 

for (1 • 0? 1 < nil? 1**) 

t*-(ic3 (Hid] (Out] (1 J *lvl[Hid] (1J); 
for (1 • npl? 1 > 0? 1— ) 

t*» (ocl (Hid] (Out) [i-1]* (ovlfHid] (Out] (i]*ovl(Hid) (Out] [1- 

nvl (Out ]*-t? 
ovl(Mid) (Out] (0) • t? 

} /* end for Out •/ 

] /• end for Hid •/ 

/* compute sigmoid for output layer neurons */ 
for (Out • 0? Out < n2? Out**) 

nvl (Out] • sigmoid (nvl (Out I) ? 

/• Maintain the last nil inputs and outputs for Input filters •/ 
iptr - ihO(nil-l) ; 


optr • ohO(nxl-l); 

for (i - nxl-1? 1 > 0? 1—) 

l 

ihO (1) • ihO(l-l)? 
oh0(i] - ohO (1-1 ]? 

] /• end for 1 */ 

ihO (0 } - iptr? 

oh0(0] - optr? 

for (In • 0? In < nO? In**) 

( 

for (1 - 0? 1 < niO? i**) 

lh0(0) (In] (1} - lvO(In) (1] ; 
for (Hid - 0; Hid < nl; Hid**) 

for (i - 0? 1 <- npO? 1**) 

oh0[0] (In] (Hid] (11 - ovO(In) (Hid] (11? 

] /• end for In */ 

) /• end PropagateForward */ 


void ComputeError (Sample, dymax, esum) 
int Sample; 
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float *dymax, *asua; 

,( 

int Out; 

static float dyabs; 
for (Out • 0; Out < n2; Out**) 
i 

Out Error (Out) • TastWavaOtft (Out | (Sampla) - nvl(Out); 

•«*um (dyabs-fabs (OutError (Out) ) ) ? 
if (dyabs > • dymax) 

•dymax - dyabs? 

I /• and Out */ 

1 /* and CoaputaErrer */ 

void Propagatalackward (Saapla, dymax, asun) 
int Samp la? 
float 'dymax, *asu»? 
f 

Int 1, In, Hid, Out; 
static float dyabs; 

/• first computa -tha arror at tha output layar •/ 
for (Out - 0; Out < n2? Out**) 

( 

Out Error (Out) - 

WavaOut (Out) (Saapla) - nvi(Out) ; 

«asua (dyabs«f aba (OutError (Out})); 

if (dyabs > *dyaax) 

•dymax * dyabs; 

0 ut Error ( Out )• »d_s i gmo i d ( nv l ( Out )) ; 

) /• and for Out •/ 

/* now computa waigftt changa for naurons in tha hiddaa to output layar 
and aaka tha corractlons - / 

for (Out • 0; Out < n2? Out*-*) 

for (Kid • 0; Kid < nl; Kid**) 

gradlant (did (Hid) (Out }, ivl(Hid), 
docKKtdl (Out |, ovi(Kid( (Outl, 
nxl, npl, Outfrror(OutJ); 
for (Out - 0; Out < n2; Out**) 

for (Hid • 0; Kid < nl; Hid**) 

»PPly_corraction del, 
dlclrocl, 

docl,aicl, mod. Hid, Out, nsl, 
npl); 

for (i - 0; i < nil? 1**) 

I 

for (Hid • 0; Hid < nl? Hid**) 

{ 

HldError (Hid } • 0.0; 

for (Out • 0; Out < n2; Out**) 

HidError (Hid]*-(icl (Hid) (Out) (1) ‘OutError (Out)) ; 
HldError (Hid) — d jdgaoid (ivl (Hid) (i) ) ; 
for (In • 0; In < nO? In**) 

gradlant (dicO(In) (Hid), ih0[i)(ln], 
doeO(Zn) (Hid) , oh0(l) (In) (Kid), 
nzO, npO, HldError (Hid) ); 

1 /• and for Hid •/ 

} /• and for l •/ 

for (In • 0; In < nO? In**) 

for (Hid • 0; Hid < nl; Hid**) 

*PPly_corraction (icO, 
dTc0,oc0, 

doc0,ale0, nocO, In, Hid, 
nsO, npO); 

) /* and PiopagataBackward */ 

void laarn () 

( 

int Out, i, 3, n - 50, TastNatwork - SO, TastErrNatvork • 10, 

Savawts * 50; 

float dymax, asum, TastKax, TastSum, high • 0.0, low • 0.0; 

FILE *ErrorFila, *Tastrila, •OuikTastfila; 
string Wtsrila; 

prlntf C Filanama to stora gulck tatt parformanca rasulta> •); 
if (strlan (gats (strl) } } 

OuikTastrila * fopantstrl, •*•); 

alsa 

QuikTaatrila - HULL; 

' prlntf {• Filanama to stora dataiJad tast parformanca rasults> •); 
if (atrlan(gats (stri) )) 

TtstFila - fopan(atrl, •w*)? 

alsa 

TastFlla - HULL; 

prlntf (* Flla to stora arror functiorv> "); 
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If (strlen (gets (strl) ) ) 

ErrorFile - fopen(strl, •«*); 

•1st 

Errornit - NULL? 

printf <“ File to ssv« weights to> "); 
if (strlen (gets (WtsFlle) ) ) 

< 

printf ("Feriodlcally save Mights every Id passes >•, SaveNts) ; 
sscanf (gets (strl), Md“, iSaveWts); 

) /• end Ntsriie •/ 
get_l© 0; 

printf (• gat weights fro* fii«> •); 
if (strlenlgets (strl) )) 

get_coeff iciants (strl) ; 

printf (■Farfont quick network performance ease every Id pessas> * 

, Test ErrNst work) ; 

sscanf (gats (strl) , Md\ iTescErrNetwork) ; 
printf ("Farfoni detailed network performance test every Id passes (must be multi 
pie of qulek network test)> • , last Network) ; 

steanf (gats (strl), Md“, iTestNetwork) ? 

printf (* range for input noise <11, 2f 11.20 ", low, high); 
sscanf (gats(strl), MM f, alow, ihigh); 

printf (• cycles to process, learning rata and momentum constant <!d II. If lt.2f> 

m 

# 

n, alpha, delta); 

- while (getchO t« 'q') 

{ 

sscanf (strl, Mdlflf, in, lalpha, idelta); 
epsilon • 1.0-delta? 
for (i - 0; i < n? i++) 
i 

if ( ({11 SaveWts) — 0) i( (strlan (WtsFlle) ) ) 
sava^coaff iciants (WtsFlle); 

esurn - dymax - 0.0; 

for (j * 0; 3 < (NumSaaplas * NumSets); 3++) 

( 

If ((31 NumSamples) •* 0) 
ra set_ne twork () ; 

FropagataForward (3, low, high, Wavaln) ? 

FropagataBackward (3, idymax, tesum); 
fflush (ErrorFile) ; 
total_cyclea++; 

) /* and for 3 •/ 

if ( (TestErrNatwork !- 0} n ((11 Test ErrNat work) — 0)) 

( 

TastMax - TestSum * 0.0; 
reset network (); 

if { (TestFlle !- NULL) II ( (i % TastHatwork) — 0)) 
fprintf (Tastrila, “Id \n*,i); 
for (3*0; 3 < (NuaTastSets * NuaTest Samples) ; }+♦) 

( 

if ((3 I NuaTest Samples) — 0) 
reset ^network i) ? 

Propagate Forward (3, low, high, TestWaveln) ; 

CoaputaError (3, ilastHax, Hast Sum) ; 
if ((TestFlle !- HULL) II ({II TestNetwork) — on 
( 

for (Out • 0; Out < n2; Out**) 

fprintf (TastFila, Ml.2f \f, nvl(Out)); 
fprintf (TastFila, •\n"}; 

1 

) /* and 3 */ 

if (QuikTastrile HULL) t 

fprintf (OuikTastriia, Md \t 16. 2f \t 16. 2f \n«, 

1, TastKax, 

TastSum/ (float) (n2*Nu»TestSamplesi ) ; 
fflush (OuikTastriia); 

) /• and TastFila •/ 

} /• and TastNatwork •/ 
if (ErrorFile !• NULL) ( 

fprintf (ErrorFile, ■ 16. 2f Nt %6.2f\n*, 

dymax, asu*/ (float) (n2 ‘NumSamples * NuaSats)); 
fflushtErrorFila) ; 

) 

) /• and for 1 •/ 

printf ("max error • 16. 2f average error • li.2fVn H , 
dymax, esua/ (float) (n2*Nu»Sample**NumSets) ) ? 
printf (" cycles to process, learning rata and momentum constant <ld 16. 
If I6.2f> or <q> to quit >•, 

n, alpha, delta); 

) /* and while */ 

/• printf (* save coefficients to fila> •)? 

If (strlentgets (strl) ) ) 

save coefficients (strl); */ 
if (ErrorFile ! - NULL) 

feiose (ErrorFile); 

If (TestFlle !- NULL) 

feiose (TestFlle); 



33 


5,253,329 


34 


) /• and laara V 


void PropOnly () 

* 

int i, 3, k»0, n • 50; 

float max^arrcr^valua, ava^arror^valua, d; 

float high • 0.0, low - 0.0; 

FILE *RasuitsFila, *arror_raeord_fila; 


131 ); 


printf (■ Fila to stora propagation ra*ult*> •); 
if (strlan (gats Utrl) ) ) 

RasultaFila - fopan(strl, •«•); 

aisa 

RasultaFila - NULL; 

printf (* Fila to rscord hiddan activations? > -J; 
if (atriantgata (strl) )) 

HidFila - fopan{atrl, ••*-); 

alsa 


HidFila * NULL; 

printf (" fila to atora racord of natwork arrors> ■); 
if (strlan (gats (atrl) ) ) 

arror racord fila • fopantatrl, ■*■); 

alaa 


arrcr_racerd_fila - NULL; 
gat_io O ; 

printf (" gat waights fcoa fila> •); 
if (strlan (gats (strl) ) 1 

gat_coaff iciants (strl) ; 

printf (* ranga for input noisa <%l.2f %6.2f> •, low, high); 
sscanf (gats (strl), *%f%f”, slow, thigh); 
if (RasultaFila) 

( 

for (i • 0; i < nO; i++) 

fprlntf (RasultaFila, •\twtvaln(*d)*, i); 
for (i • 0; i < n2; i++) 

fprlntf (RasultaFila, -UWavaOut [%d] *•, i); 
fprintf (RasultaFila, •\nVn") ; 

1 /* and if •/ 

printf (• cyclas to procasa <%d > ",n); 
whila (gatcht) !- 'q') 
i 

sscanf (strl, Md", *n); 
for (i -0; i < n; i«-+) 

( 

for (j*0; j < (NumSaaplts • NuaSats); 3+*) 
i 

if ((3 % NumSamplts) •• 0) 

( 

if (RasultaFila) 

putc( # \n' f RasultaFila); 
rasat_natwork (); 

) /• and if *7 

PropagataForward (3, low, high, Navaln) ; 
if (RasultaFila !*-NULL) 


( 


for (k * 0; k < nO; k**) 

fprintf (RoeuitsFUo.-Nt 


wavaln(k) 


for (k • 0; k < n2; k**) 

fprintf (RasultaFila, -\t nvllklJ 


f 


_arror_valut) 


valua, ava_arror_valua) 


puce (' \n' , RasultaFila) ; 

) /• and if •/ 
if (arror racord fila) 

( 

aux_arror_valua » ava_arror_valua - 0.0; 
for”(k » 0; k < n2; k++) " 

( 

if ( (d*f abs (NavaOut ( k ) ( j J -nvl ( k )) ) > max 

*ax_arror_valua * d; 
ava_arror valua+*d; 

) /• and for k */" 
ava_arror_valua/-n2; 

f print f (arr or_raeord_f i la , ■ %f \t % f \n ■ , aa x_arror_ 

} /* and if •/ 

) /* and for 3 */ 


If (RasultaFila) 

putc (' \n' , RasultaFila); 
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) /* end for i V 

prlntf (• cycles to process<%d> or <q> to quit> ", n); 
} /* «nd while •/ 
if (Hidrilo) 

fclose (HldFile) ; 
if (ResultsFile) 

fclose (ResultsFile); 
if (error_recerd_file) 

fcioso (error_record_file); 

) /• ond FropOnly */ 

/* void impulse () 
l 

int i, n; 

float *temp, "FFTReal, • FFTImaginary, result; 

FILE "f; 

init_trig (rrTSixe); 

rrTReal - (float*)*aa_alloc{FFTSite*sixeof (float) ) ; 

m Imaginary - (float*) mea_alloc(rrTSlxe"sixeof (float) ) ; 

prlntf ("Enter layer designation < 0- Hidden Layer, 1-Output Layer>"); 

prlntf (• value of initial impulse> ") ; 

input j value [0] - atof (gets (strl) ) ? 

print? ("Spectral Inpulse Response file nane> 

if (atrlen (gets (strl) ) ) 

f • f open (strl, *w«); 

prlntf (" periods to propagate lnpulse> "); 
n * atol (gets (strl) ) ; 

Reset network 0; 
while (getchO 1* 'q'j 
t 

if (strl (0) — '0') 

( 

fprintf (f, "Input To Kidden Spectral Impulse Response")? 
for (Kid - 0? Mid < nl? Kld++) 
l 

fprintf ("Spectral Impulse response for hidden node %d", Mid); 

for (1 - 0? i < rrTSixe; 

rnualUI • rrrinaginaryUl • 0.0; 

for (In ■ 0/ In < nO; In**) 

for (pulse • 0; pulse < n; pulse**) 

rrTReal [(FFTSixe/2) ♦pulse) ♦- 

new^output (IcO, IvO, 

oo0,ov0, 

In, Hid, nsO, npo, 
input value (pulse]) ? 

for (I - FFTSite/2, n • (FFTSixe/2) -1; 1 < FFTSixe; l**, n— ) 

FFTReal (n J - rrTReal (11; 
f ftl (FFTReal, FFTImaginary, rrTSixe, 7) ; 
for (i - 0? 1 < rrTSixe; i**> 
fprintf (f , " %10.2f \n", 

sgrt (FFTReal [1) * FFTReal (i ) + FFTImaginary [i) "FFTImaginary (1 )) ) ; 

) 

) 

else 

If (strl (0) — '1') 

KldToOutlmp (); 

else 

if (strl(0) — '2') 

All Net Imp (); 

HilnMenu () ? 

temp - (float*)mem allocUnum inp co«ff*num_pol*s*l) "sixeof (float) ) ; 
for (i - 0; i < rrTSixe; !*♦)* 

FFTReal (1 } - FFTImaginary (i I - 0.0; 
for (i - 0; i < nua inp coeff; i**l 
( 

teap(i] - lnput_value (ij ; 
input valued] - 0.0; 

) 

for (1 - 0; 1 <• nua poles; i**) 

( 

tempd + nua_inp_cotf f } • output a _value(l] ; 
output valued J • 0.0; 

I 


prlntf ("Spectral lapulse Response file n*me> 
if (strlen(gets (strl) ) ) 

( 

f - fopen(strl, "w"); 
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printf {■ period* to propagit* inpul«#> 
ft - *toi (got* (itrl) } ; 
fprintf <r, * Inpul so ro«pon*o\n*> ; 
for (1 • 0; i < n; i+*) 

< 

fprintf (f, * IX0.2f\n", r«*ult*n«w output (0.0) ) ; 
rm«*iurrTSizo/2)*i] - rtiuit; 

) 

\ 


for {1 - FFTSlio/2, n - <FFTSix«/2) -1? i < FFTSiit; t++, n--> 

FFTRoal (n ) - mRMlUl; 

fftl{Fm«*l, FFT Imaginary, FFTSixt, 7j ; 

fprintf {f, "Fourlor Transform of Impulse R**pcn**\n") ; 

for it - 0; 1 < rmixo; i+*) 

fprintf (f, M10.2f \t %10.2f \t %10.2f \n-,FFTR«im, FFTImaginaryti), 
■qrt (FFTRea 1(11 *FfT Reel (11 4 FFTImaginary [i]*FFTI«aginaryUl> ) ; 


for (1 • 0; 1 < nua_inp_co«ff; 

inputjulu#(l| • toSpllT; 

for (1 • 0; 1 <• num_pol««? !♦♦) 

output_v*lu«(i| * tompU + nua^lnp^coeril ; 

free (ttap) ; 

free (f rTReal) ; 


free {FTTI magi nary) ; 

fclose (fj ; 

printf (■ optlon> •); 

) */ 

void MainManu () 
l 

printf CNnMEHU-); 

printf {"Vnieern from training file <!>•); 
printf {"Vnpropagata fro» file <p>"); 

/• printf (■\nspectr»l impulse response<l>") ; •/ 
printf {"\ndo nothing - quit<q>*); 
printf ("\ftaction? >*); 

* 

main 0 
{ 


alloctte_netvork ()? 

MainMenu”() ; 

while (gatchO !- 'q') 

( 

if (strl [0] •• '1') 
learn (); 

else 

if (strltOl — 'pM 
PropOnly 0; 

else 

/* if (strl[0) — * 1' ) 

impulse O; •/ 

MainMenuU ; 

}/• end while •/ 

1 /• end main */ 


What is claimed is: 

1. A processing element (i) for use in a space-time 
neural network for processing both spacial and tem- 
poral data, wherein the neural network comprises a 
plurality of layers of said processing elements, the plu- 
rality of layers comprising a first layer and at least one 
additional layer, the network further comprising con- 
nections between processing elements of the first layer 


and processing elements of an additional layer: each 
said processing element adapted to receive a sequence 
of signal inputs X(n), X(n — 3), X(n — 2) . . , , each input 
X(n) comprising K signal components xi(n), X 2 (n), . . . 

65 x/n), . . . x*(n), each said processing element compris- 
ing, in combination: 

(a) a plurality K of adaptable filters (Fj /, F 2 /, . . . Fy/, 
. . . F ki) each filter Fy/ having an input for receiving 
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a respective component x/n), x/n — 1), x/n — 2), . . 

. , of said sequence of inputs, where x/n) is the most 
current input component, and providing a filter 
output y/n) in response to the input x/n) which is 
given by: 

y/n)^/a mJ Y/n-m), b^jX/n-k)), 

where a m j and b kj are coefficients of the filter F )/ and f 
denotes the operation of the filter; 

(b) a junction, coupled to each of said adaptive filters, 
providing a non-linear output p,<S/(n)) in response 
to the filter outputs y/n) which is given by: 

piStn))=fiyjtn)\ 
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and bfy of each filter Fy/in dependence upon the junction 
output p,{S/(n)). 

13. The processing element defined in claim 12, 
wherein said adjusting means includes means for deter- 

5 mining an error in the output p/(S/n)) between the ac- 
tual and desired response of the processing element (i) 
and adjusting the filter coefficients a m y and b*y of each 
filter Fji in dependence upon said error. 

14. The processing element defined in claim 13, 
10 wherein the non-linear transformation is a sigmoid 

transfer function with output p^S^n)) given by: 

15. The processing element defined in claim 14, 
15 wherein said error A/(n) is given by: 
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where Sj(n) is the sum of the filter outputs, 

whereby said junction presents a sequence of out- 
put signals, pXS/(n)), p,<S/<n-l)), p/(S;<n-2)). 

2. The processing element defined in claim 1, wherein 
said non-linear output provided by said junction is the 
sum SXn) of the filter outputs modified by a non-linear 
transformation p,(S,<n)) to the sum S/<n), where S/<n) is 
given by: 

Sin)=2jyj(n). 

3. The processing element defined in claim 2, wherein 
the non-linear transformation is a sigmoid transfer func- 
tion given by: 

pX 5X«»-= l/d +*—««))- 

4. The processing element defined in claim 1, wherein 
said filters are non-linear filters. 

5. The processing element defined in claim 4, wherein 
said non-linear filters are exponential auto-regressive 
filters. 

6. The processing element defined in claim 1, wherein 
the coefficients a m y and b kj of each filter Fy/ are adjust- 
able. 

7. The processing element defined in claim 1, wherein 
said adaptable filters are digital filters. 

8. The processing element defined in claim 7, wherein 
said filters are linear filters. 

9. The processing element defined in claim 8, wherein 
said filters are recursive, infinite impulse response filters 
and wherein the response of each filter is given by: 


Ltn)=mn)-Atn)) p\Sm 
where: 

D/(n) is the nth desired response from a given se- 
quence for neuron i at the output layer 
A/(n) is the network’s output response i for the nth 
input sequence pattern 

p'(S^n)) is the first derivative of p/(S/(n)), the non-lin- 
ear transfer function for the ith output’s activation 
value or in the case of said sigmoid non-linear 
transfer function, p'(S/(n)) is given by: 

p'{Stn))=ptSinW -ptfin))). 

16. The processing element defined in claim 15, 
wherein said filter coefficient b ,y* is adjusted in accor- 
dance with the formula: 

Lbijk^a[T}Lbijk old +{\ -T))Wn) xj ( n-k )] 

where: 

Aby* is the update for a zero coefficient, b //*, lying 
between processing elements i and j 
a is the learning rate of the neural network 
Ab ijk°! d is the most recent update for the kth zero 
element between processing elements i and j 
t ) damps the most recent update 
X/n — k) is the output of the jth neuron in the hidden 
layer. 

17. The processing element defined in claim 15, 
wherein said filter coefficient a is adjusted in accor- 
dance with the formula: 
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M 

y/n) - 1 a m jy/n - m). 

m = 1 

10. The processing element defined in claim 8, 
wherein said filters are nonrecursive finite impulse re- 
sponse filters and wherein the response of each filter is 
given by: 

N 

y/n) = J bkpc/n - k). 

11. The processing element defined in claim 8, 
wherein the response of each filter is given by: 

M N 

yin) - 1 a m jy/n - m) + 2 b k jx/n - k). 

m= 1 k — 0 

12. The processing element defined in claim 11, fur- 
ther comprising means for adjusting the coefficients a m j 


bajjk~a[T)Laijk old + 1 -tj)A in) y/n —A)] 
where: 

Aa//* is the update for a pole coefficient, a jjk, lying 
between processing elements i and j 
a is the learning rate of the neural network 
Aa ijk old is the most recent update for the kth pole 
coefficient between processing elements i and j 
7] damps the most recent update 
y,/n — k) is the activation value for the filter elements 
between neurons i and j, k time steps ago. 

18. The processing element defined in claim 15, 
wherein said filter coefficients a p and b,y* are adjusted 
in accordance with the formula: 

A^jfe=a[TjAi/yft oW +(l -*r))AX») x j C n ~k )] 

La ijk - a[r\Laijk old +{\ - tj)A in) y { j (n - k)] 

where: 
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Aa ijk is the update for a pole coefficient a lying 
between processing elements i and j 
. Ab ijk is the update for a zero coefficient by* lying 
between processing elements i and j 
a is the learning rate of the neural network 5 

Ab ijk old is the most recent update for the kth zero 
element between processing elements i and j 
7] damps the most recent update 
Aay* oW is the most recent update for the kth pole 
element between processing elements i and j 10 
x/n— k) is the output of the jth neuron k time steps 
ago 

y//(n— k) is the activation value for the filter element 
between neurons i and j, k time steps ago. 

19. A neural network for processing both spacial and 15 
temporal data, wherein said neural network comprises a 
plurality of layers of processing elements, the plurality 
of layers comprising a first layer and a second layer, the 
network further comprising connections between pro- 
cessing elements of the first layer and processing ele- 20 
ments of the second layer; said first layer of said net- 
work adapted to receive a sequence of signal inputs 
X(n), X(n — 1), X(n — 2) ...» each input X(n) comprising 

N signal components xi(n), X 2 (n), . . . x/n), . . . XA<n), 
said first layer of said network comprising, in combina- 25 
tion: 

(a) a plurality L of first processing elements, each first 
processing element (i) comprising a plurality N of 
adaptable filters (Fi/, F 2 /, . . . F //, . . . F tv/), each 
filter Fjf having an input for receiving a respective 30 
component x/n), x/n— 1), x/n — 2), . . . , of said 
sequence of inputs, where x/n) is the current input 
component, and providing a filter output y/n) in 
response to an input x/n) which is given by: ^ 

yj(n) =Aa m jyfn - m), bkjxfn - k)\ 

where a m j and b kj are coefficients of the filter F/y and f 
denotes the action of the filter; 

each first processing element (i) further comprising a 40 
first junction, coupled to each of said adaptive 
filters, providing a non-linear output p,<S,<n)) in 
response to the filter outputs y/n) which is given 
by: 

45 

Pi(Si(n))=Ayj(n)), 

where S/{n) is the sum of the filter outputs, 
each first junction presenting a sequence of first out- 
put signals, p,<S,<n)) f p,<n- 1)), p i {S / <n~2)), 50 

20. The neural network defined in claim 19, wherein 
said second layer comprises: 

a plurality of M of second processing elements (k) 
each coupled to a plurality of said first junctions, 
each second processing element comprising a plu- 55 
rality O of adaptable filters (F 1 *, F 2 *, . . . F /*, . . . 
Foa:), each connected to one of said first junctions, 
each filter F hk having an input for receiving a re- 
spective first junction output signal S*(n), Sjt(n — 1), 
Sk(n— 2), . . . , of said sequence of first junction 60 
output signals, where S*(n) is the most current 
output signal, and providing a filter output, y/t(n), 
in response to an input S^n) which is given by: 

y&)=AC gk Y h (n-I)), d rkPk (Sk(n-r)), 65 

where C q k and d r k are coefficients of the filter F hk and f 
denotes the action of the filter; 
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each second processing element (k) further compris- 
ing a second junction, coupled to each of said sec- 
ond adaptive filters of the respective second pro- 
cessing element and providing a non-linear output 
Pg(Sg(n)) in response to the filter outputs y*(n) 
which is given by: 

Pg(S^n))=AyH(n)), 

where Sg(n) is the sum of said second filter outputs, 

each second junction presenting a sequence of second 
output signals p*(S^n)), pg(S*(n- 1), p g (S g (n-2)), . 

21. The network defined in claim 20, wherein said 
non-linear outputs provided by said junctions are a sum 
Sg(n) of the filter outputs modified by an arbitrary non- 
linear transformation pg(Sg(n)) to the sum Sg(n), where 
Sg(n) is given by: 

•%(») - 2 yh (n). 

22. The network defined in claim 21, wherein the 
non-linear transformation is a sigmoid transfer function 
given by: 

pXSX*))=l/(l+e-SX«)>. 

23. The network defined in claim 20, wherein said 
filters are non-linear filters. 

24. The network defined in claim 23, wherein said 
non-linear filters are exponential auto-regressive filters. 

25. The network defined in claim 20, wherein said 
adaptable filters are digital filters. 

26. The network defined in claim 25, wherein said 
filters are linear filters. 

27. The network defined in claim 26, wherein said 
filters are recursive, infinite impulse response filters and 
wherein the response of each filter is given by: 

M 

yfti) -- Z Cmjyfti - m). 
m=\ 

28. The network defined in claim 26, wherein said 
filters are non-recursive finite impulse response filters 
and wherein the response of each filter is given by: 

N 

yfa) = ^ d k jxj(n - k). 

29. The network defined in claim 26, wherein the 
response of each filter is given by: 

M N 

yft 1 ) = £ c m pfn - m) + Z dkjxfn ~ *)■ 

m — 1 Ac — 0 

30. The network defined in claim 29, wherein the 
coefficients c m j and d kj of each filter F ),• are adjustable. 

31. The network defined in claim 29, further compris- 
ing means for adjusting the coefficients c m j and d kj of 
each filter F/y in dependence upon the plurality N of 
junction outputs pg(Sg(n)). 

32. The network defined in claim 31, wherein said 
adjusting means includes means for determining and 
error in said outputs p/Sg(n)) between the actual and 
desired response of the network and adjusting the filter 
coefficients c m j and d kj of each filter F )/ in dependence 
upon said error. 
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33. The network defined in claim 32, wherein the 
non-linear transformation is a sigmoid transfer function 
given by: 

/>/*/*))= 1/(1 5 

34. The network defined in claim 33, wherein said 
error is given by: 

10 

where: 

D/n) is the nth desired response from a given se- 
quence for neuron g at the output layer 

A/n) is the network’s output response at neuron g for 
the nth input sequence pattern 

p'(S/n)) is the first derivative of the non-linear trans- 
fer function for the gth output’s activation value or 
in the case of said sigmoid non-linear transfer func- 
tion, p'(S/n)) is given by 20 

(1 —p)Sj£n))). 

35. The network defined in claim 34, wherein the kth 
zero coefficient d y* of the filter between first processing 
element j and second processing element i is adjusted in 
accordance with the formula: 
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Ac ijk is the update for the kth pole coefficient cy* 
lying between first processing element j and second 
processing element i 

Ady* is the update for the kth pole coefficient d ijk 
lying between first processing element j and second 
processing element i 

a is the learning rate of the neural network 
Ad ijk old is the most recent update for the kth zero 
element between first processing element j and 
second processing element i 
7] damps the most recent update 
Ac ijk old is the most recent update for the kth zero 
element between first processing element j and 
second processing element i 
x/n— k) is the output of the jth first processing ele- 
ment k time steps in the past 
y,/n — k) is the activation value for the filter element 
between first processing element j and second pro- 
cessing element i, k time steps in the past. 

38. The network defined in claim 34, wherein the kth 
pole coefficient for said filter between network input 
element j and first processing element i, a//*, is adjusted 
in accordance with the formula: 

Aa ijk =a[r)Aajj k old +0 -17)^,/*-*)] 


Ad iJk = a [T]Adij k old +( 1 - 17) A «/*-*)] 


where: 

Ady* is the update for a zero coefficient, dy*, lying 
between first processing element j and second pro- 
cessing element i 

a is the learning rate of the neural network 
Ad ijk oId is the most recent update for the kth zero 
coefficient between first processing element j and 
second processing element i 
17 damps the most recent update 
x/n — k) is the output of the jth first processing ele- 
ment k time steps in the past. 

36. The network defined in claim 34, wherein the kth 
pole coefficient for said filter between first processing 
element j and second processing element i, cy*, is ad- 
justed in accordance with the formula: 
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Acij k =a{r\Acijk old +{\-r\) A ,>/*-*)] 


where 

Ac ijk is the update for the kth pole coefficient, c y*, 
lying between first processing element j and second 5 
processing element i 

a is the learning rate of the neural network 
A Cijk old is the most recent update for the kth pole 
coefficient between first processing element j and 
second processing element i " ' 

rj damps the most recent update 
y,/n — k) is the activation value for the filter element 
between first processing element j and second pro- 
cessing element i, k time steps in the past. 

37. The network defined in claim 34 wherein said 
filter coefficients cy* and dy* are adjusted in accordance 
with the formulae: 


Ad ijk =a[j}Adijk old + ( 1 - V) A*/* - *)] 
Ac ijk = aft Aqjk old + (1-tj) Ay,,/* - k)} 


where 

Aay* is the update for the kth pole coefficient, ay*, 
lying between network input element j and first 
processing element i 

a is the learning rate of the neural network 
AsLijk oId is the most recent update for the kth pole 
coefficient between network input element j and 
first processing element i 

€/ is the backpropagated network error at the ith first 
processing element 
rj damps the most recent update 
y/n — k) is the activation value for the filter element 
between network input element j and first process- 
ing element i, k time steps in the past. 

39. The network defined in claim 34 wherein said 
filter coefficients ay* and by* are adjusted in accordance 
with the formulae: 

A bijk = a[r\Abijk oId + ( 1 - t fiepcjin - *)] 

Aaijk= a[y)Aaijk old +{\ - 17 )^/* - *)] 

where: 

Aay*is the update for the kth pole coefficient ay* lying 
between network input element j and first process- 
ing element i 

Aby* is the update for the kth zero coefficient by* 
lying between network input element j and first 
processing element i 

€/ is the backpropagated network error at the ith first 
processing element 

a is the learning rate of the neural network 
Ab ijk old is the most recent update for the kth zero 
element between network input element j and first 
processing element i 
77 damps the most recent update 
Aa ijk old is the most recent update for the kth pole 
coefficient of the filter between network input 
element j and first processing element i 
x/n— k) is the jth network input k time steps in the 
past 


where: 
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y//(n — k) is the activation value for the filter element 
between network input element j and first process- 
ing element i, k time steps in the past. 

40. The network defined in claim 34, wherein the kth 
zero coefficient by* of the filter between network input 5 
element j and first processing element i is adjusted in 
accordance with the formula: 

bbijk=a[T)bbijk old +{\ -T))epr/rt-*)l 

10 

where: 

Aby* is the update for a zero coefficient, by*, lying 
between network input element j and first process- 
ing element i 

€/ is the backpropagated network error at the ith first 15 
processing element 

a is the learning rate of the neural network 

Ab ijk old is the most recent update for the kth zero 
coefficient between network input element j and 
first processing element i 20 

tj damps the most recent update 

x/n — k) is the jth network input k time steps in the 
past. 

41. The network described in claim 33, further com- 
prising a means for propagating the error Ag(n) mea- 25 
sured at the outputs of the gth second processing ele- 
ment backward through the intervening filter connec- 
tions between first and second processing elements 
thereby to provide a means for adjusting the coeffici- 
ents of the filters which connect the inputs of the net- 30 
work to the first processing elements. 
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42. The network defined in claim 38, wherein said 
means for backward propagation of error is described 
by the formula: 


€*«) = P'KSXn)) £2 ^2 o djik^fn + *) + J J c jiWifn - *) J 

where 

cXn) is the result of backward propagation of network 
error from the outputs of all second processing 
element through the filters between first processing 
element i and the plurality N of second processing 
elements 

Cjik is the kth pole coefficient of the filter between first 
processing element i and second processing ele- 
ment j 

dy* is the kth zero coefficient of the filter between 
first processing element i and second processing 
element j 

T and U are respectively the non-recursive and recur- 
sive orders of the filter through which back-propa- 
gation occurs 

A/n +k) is the error computed at the output of the jth 
second processing element k time steps in the fu- 
ture 

7y(n— k) is the output from k time steps in the past of 
the filter operating on the inverted sequence of 
network errors. 

4 4 4 4* 
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